Appendix G — Practice Final Solutions

STAT303-2 (Winter2023)

Abstract

These solutions are being composed by students of the course STAT303-2 (Winter 2023).

G.1 Potential problems

Presence of which of the following potential problems in a linear regression model may lead to statistically significant variables appearing insignificant?

1. Multicollinearity

2. Outliers

3. Overfitting

Answer: A and B

Explanation:

A) Multicollinearity:

Recall, the estimated variance of the coefficient \(\beta_j\), of the \(j^{th}\) predictor \(X_j\), can be expressed as:

\[\hat{var}(\hat{\beta_j}) = \frac{(\hat{\sigma})^2}{(n-1)\hat{var}({X_j})}.\frac{1}{1-R^2_{X_j|X_{-j}}}\]

If the predictor \(X_j\) is collinear with other predictors, \(R^2_{X_j|X_{-j}}\) will be large, which in turn will inflate \(\hat{var}(\hat{\beta_j})\). In other words, multicollinearity inflates the standard errors of the coefficients for which the variables are collinear. Since \(t\)-statistic is calculated by dividing the estimated coefficient by its standard error, the \(t\)-statistics shrinks, and the corresponding \(p\)-value increases. Therefore, the hypothesis test loses the power to reject the null hypotheses, and thus statistically significant variables appearing insignificant.

Another way to think about this can be that if some predictors are collinear, it can be difficult to separate out the individual effects of these variables in the response and significant variables may appear insignificant.

B) Outliers

Recall, the estimate of error variance is given by:

\[\hat{\sigma}^2 = {\frac{RSS}{n-2}},\] where RSS is the residual sum of squared errors. Outliers result in an increase in \(RSS\), leading to an increase in the estimated error variance \(\hat{\sigma}^2\), which in turn inflates \(\hat{var}(\hat{\beta_j})\). The rest of the explanation follows from the previous explanation on multicollinearity.

C) Overfitting

Overfitting shrinks \(RSS\), which in turn shrinks \(\hat{\sigma}^2\), thereby shrinking \(\hat{var}(\hat{\beta_j})\). Thus overfitting will act in way opposite to what we observe in (A) and (B).

G.2 Potential problems

Classify a data point as influential / outlier / high leverage in a linear regression model, based on the description.

1. The data point is likely to have a large effect on the model in terms of prediction: Influential point

2. The data point has the potential to have a large effect on the model in terms of prediction: High leverage point

3. The data point is likely to inflate the model R-squared: High leverage point that is not influencial

4. The data point is unlikely to have a large effect on the model in terms of prediction: outlier

Explanation:

See the graphics in class presentation on Chapter3_Outliers_high_leverage_influential_points. Think of influential points / high leverage points / outliers as a force (proportional to the residual corresponding to the point) pulling a canteliver beam. Depending on the position from where you pull the cantilever beam, you may move it too much or too little.

A) Inluential point (high leverage & outlier): an outlier with the respect to both the predictor and the response. It has a large effect on the regression line. As shown in the graphics, influence is higher for more extreme outliers with same leverage and for points with higher leverage & similar outlying distance.

B) High leverage point: Observations with high leverage have an unusual value for the predictor (ie. lie outside the domain of most points). High leverage point has the potential to have a large affect on the regression line. It is cause for concern if the least squares line is heavily affected by just a couple of observations, because any problems with these points may invalidate the entire fit.

C) If you have a high leverage point that is not influencial: The variance of the response may increase in the presence of high leverage points, since an unusual set of predictor values may correspond to an unusual response, which may increase the total variation. However, as the point is not inluential, the increase in the unexplained variation (the squared residual) will not be proportionate to the increase in total variation. As \(R^2\) is one minus the ratio of unexplained variation to total variation, it is likely to increase.

D) Outliers: As shown in the graphics, outliers very small effect on prediction.

G.3 Autocorrelation

A linear regression model was developed to predict the number of passengers taking a flight per month. The data consists of number of passengers flying each month from January 1949 to December 1960. The autocorrelation plot below shows the correlation of the residuals with the lagged residuals of the model. Choose the most appropriate option.

1. The above plot shows the presence of autocorrelation. The 6-month lagged response is the most appropriate lag to be added as a predictor in the model to address autocorrelation

2. The above plot shows the presence of autocorrelation. The 12-month lagged response is the most appropriate lag to be added as a predictor in the model to address autocorrelation

3. The above plot shows the presence of autocorrelation. The 1-month lagged response is the most appropriate lag to be added as a predictor in the model to address autocorrelation

4. The above plot shows the absence of autocorrelation as the plot must have a cyclical pattern in the presence of autocorrelation

5. The above plot shows the absence of autocorrelation as the one month lagged residual must have the highest correlation with the residual in the presence of autocorrelation

Answer: B

Explanation: As seen in the plot, the residuals are highly correlated (correlation of more than 60%) with lagged residuals of 12 months. This shows the presence of autocorrelation. To address autocorrelation, the 12-month laggged response will be the most appropriate as it has the highest correlation with the response. Thus, it will explain the variation in the respone the most.

There is no need for there to be a cyclical pattern for autocorrelation. Even if one of the lagged residuals are highly correlated with the residual, it shows the presence of autocorrelation.

G.4 Logistic regression (goodness-of-fit)

Which of the following metrics can be used to assess the goodness-of-fit of a logistic regression model?

1. All of these

2. LL-Null

3. Log-Likelihood

4. Df Model

5. R-squared

Answer: Log-Likelihood

Explanation In logistic regression, the response is assumed to follow a Bernoulli distribution, where the probability of success is a function of the predictors and its coefficients (the model parameters). With this assumption, one can compute the the joint probability density of the observed data as a function of the model parameters. This creates a set of probability distributions (based on different values of model parameters) that could have generated the data. The algorithm finds the values of the model parameters (the beta coefficients) such that the probability of observing the data maximizes. This probability is the likelihood, and its logarithm is the log-likelihood. The higher the log-likelihood, the more probable it is to observe the data. Thus, log-likelihood is a way to measure the goodness-of-fit of the model.

LL-NULL is the log-likelihood of the model with no parameters. This is compared with the log-likelihood of the model with predictors to test if the regression is statistically significant.

Df Model is the number of predictors in the model.

R-squared cannot be used for logistic regression as there are no residuals.

G.5 Logistic regression (threshold probability)

For a logistic regression model, as we increase the decision threshold probability,

1. None of these

2. the recall will reduce or stay the same

3. the ROC-AUC will increase or stay the same

4. the precision will increase or stay the same

5. the classification accuracy will increase or stay the same

Answer: B

Explanation: See class slide on the confusion matrix. Increasing threshold probability means that less observations are predicted to be positive. Hence, some TP could turn into FN, reducing the recall. (this might not happen if there is no observations of actual positives between the thresholds). ROC-AUC is independent of the threshold probability. Both precision and classification accuracy might decrease if the number of FP among actual negatives increase more than the increase of TP among actual positives by the shift in the threshold.